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I  ntroduction 


The  current  and  projected  Chief  of  Naval  Operations  (CNO)  and  Chief  of  Naval 
Personnel’s  (CNP)  plans  for  personnel  distribution  mandate  improvements  in  personnel 
readiness  and  services  to  the  Sailor.  The  Navy  must  seek  innovative  ways  to  improve  its 
personnel  distribution  business  practices  through  research  and  technology  to 
accomplish  this  improvement  in  readiness  and  Sailor  services.  The  current  system  is 
viewed  as  “detailer  orientated”  in  that  the  detailer,  who  has  much  less  at  stake  than  the 
Sailor  and  Navy  commands,  actually  becomes  the  primary  decision  maker.  This  has 
resulted  in  a  system  that  is  viewed,  as  documented  in  Navy  personnel  opinion  surveys, 
as  unfair  and  untrustworthy  by  many  Sailors.  It  also  results  in  less  than  optimal  skill 
matches  and  unfilled  jobs. 

With  the  advent  of  the  Internet,  the  Navy  is  exploring  a  number  of  Web  technologies 
to  improve  its  business  practices  and  make  workers  more  productive  in  the  work  place. 
The  Web-based  marketplace  is  currently  investigating  various  approaches  to 
constructing  an  electronic  marketplace  that  would  lead  to  a  more  effective  and  efficient 
distribution  and  assignment  process  for  Sailors  and  Commands.  The  purpose  of  this 
effort  is  to  support  the  Web-based  marketplace  effort  through  research  and 
development  of  intelligent  software  agents  for  use  in  bilateral  negotiations. 


The  Model 


In  matching  Sailors  to  jobs,  bilateral  negotiations  is  an  important  mechanism  to 
implement  flexible  and  distributed  matching  in  the  Navy’s  personnel  distribution 
system  (Giampapa,  Li,  Yang,  &  Sycara,  2003;  Li,  Giampapa,  &  Sycara,  2003b).  A 
negotiator,  either  a  Sailor  or  a  Command  often  has  more  than  one  potential  matching 
alternative.  This  is  especially  true  for  enlisted,  who  have  highly  valued  skills  or 
Commands  that  are  seen  as  desirable  duty  assignments.  For  example,  a  Command  may 
find  more  than  one  Sailor  who  is  qualified  for  the  job,  and  a  Sailor  can  be  informed  of 
more  than  one  job  vacancy  that  interests  him.  These  alternatives  are  called  outside 
options.  Accepting  a  proposal  in  one  negotiation  means  refusing  all  outside  options.  On 
the  other  hand  one  may  leave  a  negotiation  (called  “opt-out”  of  a  negotiation)  without 
reaching  an  agreement  based  on  the  expectation  of  reaching  a  more  favorable 
agreement  in  outside  options.  Modeling  the  outside  options  and  understanding  the 
interaction  between  outside  options  and  a  negotiation  process  is  an  essential  aspect  to 
designing  an  effective  negotiation  strategy  in  the  Navy  detailing  process. 

Outside  options  can  exist  concurrently  with  a  negotiation,  or  come  sequentially  in 
the  future  (Li,  Giampapa,  &  Sycara,  2003a).  A  concurrently  available  outside  option  is  a 
negotiation  thread  that  the  negotiator  is  involved  in  simultaneously  with  another  thread. 
This  happens  because  a  Command  may  find  multiple  potential  Sailors  who  are  available 
for  negotiations  for  the  same  job  at  the  same  time.  A  Sailor  may  also  be  invited  to  more 
than  one  negotiation— one  for  each  potential  job— simultaneously.  A  sequentially 
available  outside  option  is  a  matching  opportunity  that  comes  in  the  future.  A  Command 
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is  not  informed  at  one  time  of  all  potential  Sailors  who  will  become  available  during  the 
whole  search  period,  neither  is  a  Sailor  aware  of  all  interesting  job  vacancies  during 
their  application  period.  The  information  on  Sailors  and  jobs  that  are  available  is 
published  periodically  and  sequentially.  The  cost  of  an  information  search  may  restrict 
Sailors  and  Commands  from  having  complete  information.  Outside  options  are 
uncertain  in  terms  of  both  availability  and  quality.  The  availability  of  outside  options 
is  uncertain  because  a  negotiator  cannot  predict  the  outcome  of  a  negotiation  thread,  or 
the  preferences  of  the  other  party  in  a  negotiation  thread.  How  to  model  the  availability 
and  uncertainty  of  outside  options  is  an  important  consideration  for  modeling. 

Outside  options  affect  the  negotiation  strategies  via  their  impact  on  the  reservation 
price.  The  reservation  price  is  the  worst  agreement  that  a  negotiator  can  accept.  For 
example,  in  a  buyer-seller  negotiation  model,  the  reservation  price  of  the  buyer  is  the 
highest  price  she  is  willing  to  pay  for  the  negotiated  good.  For  the  seller,  the  reservation 
price  is  the  lowest  price  at  which  he  is  willing  to  sell  the  good.  The  price  at  which  the 
seller  is  willing  to  sell  depends  on  the  production  cost  of  the  seller.  The  price  at  which 
the  buyer  is  willing  to  buy  depends  on  the  valuation  of  the  good  to  the  buyer. 

Additionally  they  both  depend  on  the  availability  of  other  buyers  or  sellers.  From  the 
buyer’s  perspective,  if  there  are  no  outside  sellers,  the  reservation  price  is  equal  to  the 
valuation.  However,  the  negotiation  does  not  necessarily  end  with  the  reservation  price 
because  the  seller  does  not  know  the  buyer’s  reservation  price.  If  there  are  other  sellers 
joining  the  market  and  the  original  seller  is  not  a  monopoly  any  more,  then  the  buyer 
will  decrease  her  reservation  price  hoping  that  she  could  reach  a  deal  with  other  sellers, 
if  she  cannot  reach  an  agreement  with  the  current  seller  with  a  price  less  than  the 
reservation  price.  The  reservation  price  will  be  lower  if  the  buyer  expects  that  there  are 
more  outside  sellers  with  possibly  lower  prices.  Similar  positions  can  be  drawn  for  the 
seller.  The  reservation  price  impacts  the  proposal  and  response  decisions  of  a 
negotiator.  When  there  are  outside  options,  design  of  an  effective  negotiation  strategy 
can  be  divided  into  two  parts:  the  first  is  to  design  a  negotiation  strategy  given  the 
reservation  price  and  other  inputs,  the  second  is  to  calculate  the  reservation  price  based 
on  the  model  of  outside  options. 

In  our  previous  modeling  work  (Li  et  al.,  2003b)  we  have  proposed  a  nested  model 
for  negotiations  in  the  Navy  detailing  process  considering  the  uncertain  and  dynamic 
outside  options.  The  model  is  composed  of  three  modules:  (1)  a  single-threaded 
negotiations  model,  (2)  a  synchronized  multi-threaded  negotiations  model,  and  (3)  a 
dynamic  multi-threaded  negotiations  model.  These  three  models  embody  increased 
sophistication  and  complexity.  The  single-threaded  negotiation  model  provides  the 
negotiation  strategies  without  specifically  considering  outside  options.  The  model  of 
synchronized  multi-threaded  negotiations  builds  on  the  single-threaded  negotiation 
model,  and  considers  the  presence  of  concurrently  available  outside  options  by 
calculating  the  reservation  price  based  on  the  other  existing  negotiation  threads.  The 
model  of  dynamic  multi-threaded  negotiations  expands  the  synchronized  multi¬ 
threaded  model  by  considering  the  uncertain  outside  options  that  may  come 
dynamically  in  the  future. 
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Specific  solutions  of  these  models  are  presented  in  this  report.  Two  effective 
negotiation  strategies  are  proposed,  the  time-dependent  strategy  and  Bayesian  learning 
strategy.  Four  heuristic  approaches  are  designed  to  estimate  the  expected  utility  from  a 
synchronized  multi -threaded  negotiation.  A  Poisson  process  is  used  to  model  the 
random  sequential  arrivals,  and  formulas  are  provided  to  calculate  the  expected  utility 
from  a  negotiation  process  when  uncertain  outside  options  may  come  in  the  future. 
Empirical  analysis  is  provided  to  characterize  the  impact  of  outside  options  on  the 
reservation  price  and  therefore  on  the  negotiation  strategy.  The  results  show  that  the 
utility  of  a  negotiator  improves  significantly  when  she  considers  outside  options  from 
when  she  does  not,  and  the  average  utility  is  higher  when  she  both  considers  the 
concurrent  outside  options  and  foresees  the  future  ones  than  when  she  only  considers 
the  concurrent  outside  options. 

The  rest  of  the  report  is  organized  as  follows:  the  models  and  solutions,  the 
experimental  results,  and  conclusions. 

For  the  convenience  of  presentation  we  call  the  two  agents  in  a  bilateral  negotiation  a 
buyer  and  a  seller,  and  we  present  the  model  from  the  buyer’s  perspective.  The  buyer 
prefers  a  lower  value  of  the  negotiated  issue  while  the  seller  prefers  a  higher  value.  In 
the  Navy  detailing  process  we  can  regard  a  Command  as  a  buyer  and  a  Sailor  as  a  seller. 
The  roles  of  a  buyer  and  a  seller  are  interchangeable  by  changing  the  sign  of  the  value  of 
the  negotiated  issue. 

There  are  T  periods  over  the  entire  horizon  of  a  detailing  window.  Let  a  period  be 
denoted  by  t,  t  =  o...,  T  -  l.  A  buyer  needs  to  reach  an  agreement  with  a  seller  before 
period  T.  The  potential  sellers  may  unexpectedly  come  in  at  different  times  with 
different  reservation  prices,  and  the  buyer  can  negotiate  with  the  sellers  simultaneously. 
The  negotiation  between  the  buyer  and  a  seller  is  called  a  negotiation  thread.  The 
number  of  threads  in  period  t  is  denoted  by  nt,  and  the  collection  of  threads  in  period  t  is 

denoted  by  Dt  =  {di  }"'=l .  The  seller  in  the  thread  di  is  denoted  by  s;.  Based  on  the 

background  information  on  the  sellers  (or  sellers’  products),  the  buyer  can  value  the 
sellers  (or  the  sellers’  products)  differently.  For  example,  a  Command  can  attach 
different  values  to  having  the  job  filled  by  different  Sailors  based  on  the  Sailors’  skills 
and  experiences  and  the  job’s  requisites.  A  Sailor  can  also  have  different  preferences  on 
different  jobs  because  of  the  difference  in  the  location  or  properties.  Let  the  value  of  the 
seller  s,  be  Vi.  If  the  buyer  reaches  an  agreement  with  the  seller  Si  at  x,  then  the  utility  of 
the  buyer  is  Vi-x. 

The  buyer  wants  to  reach  the  lowest  possible  price  agreement  with  a  seller.  But  does 
not  know  the  bottom  line  of  the  seller,  or  what  price  is  acceptable  to  the  seller.  If  the 
price  insisted  on  is  low,  there  could  be  a  high  profit  but  risk  of  losing  the  cooperation 
opportunity  with  the  seller.  On  the  other  hand  if  the  price  she  agrees  on  is  high,  she  can 
make  a  deal  with  the  seller  with  high  probability  but  then  the  profit  is  low.  Although  the 
buyer  does  not  know  the  reservation  price  of  a  seller,  she  can  have  some  estimation  of 
the  information  based  on  statistical  aggregation  of  the  historical  data  or  survey  work. 
The  historical  data  records  the  agreements  that  were  reached  on  the  same  or  similar 
products  (jobs)  in  the  past.  A  negotiator  can  also,  maybe  by  the  help  of  a  third  party,  do 
a  survey  to  ask  the  reservation  prices  of  a  representative  population.  The  estimation  of 
the  reservation  price  of  a  seller  is  characterized  by  a  probability  distribution  F(  ),  where 
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F(x)  denotes  the  probability  that  the  reservation  price  of  a  seller  is  no  greater  than  x. 

This  probability  distribution  is  called  the  prior  belief  of  the  buyer.  A  negotiation 
provides  a  mechanism  for  the  negotiators  to  exchange  messages  and  adjust  their 
proposals.  Usually  a  negotiator  will  start  with  a  favorable  proposal,  and  then  make 
sequential  concessions  until  a  proposal  is  accepted  or  the  negotiation  deadline  is 
reached.  The  negotiation  strategy  decides  the  pace  of  concession  at  each  step  based  on 
the  single-threaded  negotiation  model  given  the  buyer’s  reservation  price  and  the  prior 
belief. 

When  there  are  outside  options,  the  decision  of  a  negotiator  is  more  complicated. 

The  buyer  will  expect  to  reach  a  utility  that  is  no  worse  than  the  expected  utility  that 
could  be  achieved  from  the  outside  options.  In  other  words,  the  buyer  has  a  threshold  on 
the  lowest  utility  that  he/she  should  achieve  from  a  negotiation  thread  based  on  the 
expectation  of  the  outside  options.  The  lowest  utility  to  achieve  in  thread  di  is  called  the 
reservation  utility  OUi  in  the  thread.  The  reservation  utility  is  equal  to  the  expected 
utility  from  the  outside  options,  which  can  be  viewed  together  as  a  multi-threaded 
negotiation.  Given  the  reservation  utility  OUi,  the  reservation  price  Ri  of  the  buyer  in 
thread  di  can  be  calculated  by  Ri  =  m  -  OUi.  If  there  are  no  outside  options,  we  can  say 
that  the  reservation  utility  is  zero,  or  the  reservation  price  Ri  is  equal  to  the  value  iy. 
Because  of  the  heterogeneity  among  the  sellers,  the  reservation  prices  in  each  thread 
may  be  different.  If  the  reservation  price  in  each  thread  is  known,  the  buyer  can  apply 
the  single-threaded  negotiation  model  to  make  the  negotiation  decisions  in  each  thread. 

Calculation  of  the  expected  utility  from  the  outside  options  depends  on  the  model  on 
the  outside  options,  and  on  the  approach  to  estimate  the  expected  utility  from  a  multi¬ 
threaded  negotiation.  In  a  synchronized  multi-threaded  negotiation  model  the  outside 
options  at  period  t  for  thread  di  are  the  other  concurrently  existing  negotiation  threads 
Di\di.  The  synchronized  model  maps  the  current  outside  options  to  the  reservation 
utility  OUi  ( Dt\di )  of  each  thread  di,  i  =  l,...,  m.  The  dynamic  multi-threaded  negotiation 
model  also  considers  the  outside  options  that  may  come  in  the  future  at  uncertain  times 
with  uncertain  values.  Let  the  probability  that  a  new  opponent  arrives  in  a  period  be  p, 
and  the  probability  distribution  of  the  value  of  an  opponent  be  ®  (•),  where  ®  (u)  is  the 
probability  that  the  value  of  an  opponent  is  less  than  v.  The  dynamic  multi-threaded 
negotiation  model  calculates  the  reservation  utility  OU(Dt\di\p,  ®  (•))  for  each  current 
thread  di  based  on  the  current  outside  options  Dt\di,  given  the  arrival  probability  and 
the  probability  distribution  of  opponents’  values.  The  dynamic  multi -threaded 
negotiation  model  can  be  viewed  as  a  synchronized  model  with  uncertain  threads. 

Next  we  first  present  the  negotiation  strategy  solution  in  a  single-threaded 
negotiation,  then  the  influence  of  the  concurrent  negotiation  threads  is  quantified  in  the 
reservation  utility.  Finally,  the  negotiation  threads  that  may  come  sequentially  in  the 
future  are  considered  additionally  and  the  impact  is  integrated  in  the  reservation  utility 
and  thus  in  the  negotiation  strategy. 
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Single-threaded  Negotiations 

We  describe  the  negotiations  based  on  an  alternating-offers  negotiation  protocol,  be¬ 
cause  (l)  it  is  a  sequential  negotiation  protocol,  which  allows  negotiators  to  dynamically 
adjust  the  offers  and  does  not  require  reasoning  and  computation  as  complicated  as  in  a 
one-shot  negotiation;  and  (2)  it  provides  more  flexibility  for  the  negotiating  parties  to 
efficiently  convey  information  than  an  ultimatum  negotiation  protocol,  in  which  one 
party  proposes  and  the  other  party  can  only  respond  by  accepting  or  rejecting  the  offers 
(Ausubel,  Cramton,  &  Deneckere,  2002;  Napel,  2002).  The  negotiation  strategy  based 
on  an  alternating-offers  protocol  specifies  the  decisions  for  both  proposal  generation 
and  response  to  an  offer. 

In  a  negotiation  following  an  alternating-offers  protocol,  the  negotiators  propose  and 
respond  alternatively,  until  one  accepts  an  offer  or  quits  the  negotiation1,  or  the 
negotiation  deadline  T  is  reached.  The  history  Hl *  of  a  negotiation  at  time  t,  t  >  1  is  a 
sequence  of  the  negotiators’  actions  before  t  (i.e.,  Hl  =  A™ m<t ,  where  A"1  is  the  action  of 

negotiator  i  at  time  m.  Therefore  the  history  of  an  alternating-offers  negotiation  at  time 
t  is  a  sequence  of  proposals  (i.e.,  Ht  =  {xla,xl,xl,x4h,...,x'a(x‘h)} ,  where  x’"  is  the  proposal 

submitted  by  negotiator  i  at  time  m.  Generally  a  negotiation  strategy  Si  specifies  the 
action  at  each  step  conditional  on  the  negotiation  history,  and  based  on  the  reservation 
price  and  prior  belief  (i.e.,  A'  SfHt  I  Ri,  Fi  (•)),  o  <  t  <  T,  where  A‘  e  {accept,  reject  and 
propose  x‘+1,  quit}.  To  give  an  optimal  negotiation  strategy,  game  theoretic  analysis  is 

required  to  derive  the  perfect  Bayesian  equilibrium  (Fudenberg  &  Tirole,  1991). 2  The 
analysis  of  the  perfect  Bayesian  equilibrium  is  not  tractable  when  both  parties  have 
incomplete  information  with  an  alternating-offers  protocol,  although  there  have  been 
conclusions  on  the  optimal  strategy  in  bilateral  negotiations  with  two-sided  incomplete 
information  in  a  direct  revelation  mechanism  when  the  prior  beliefs  of  both  parties 
follow  a  uniform  distribution  (Myerson  &  Satterthwaite,  1983).  We  adopt  two  effective 
negotiation  strategies  that  have  been  developed  in  the  Artificial  Intelligence  (AI)  domain 
and  proved  successful.  These  two  strategies  are  the  time-dependent  strategy  (Faratin, 
Sierra,  &  Jennings,  1998)  and  Bayesian-learning  strategy  (Zeng  &  Sycara,  1998).  These 
approaches  do  not  explicitly  model  the  strategic  interactions  between  the  negotiators. 
Instead  they  focus  on  some  issues  that  are  important  to  the  decision  and  information, 
and  provide  flexible  heuristic  decision  functions. 

Time-dependent  Approach 

The  time-dependent  approach  focuses  on  the  impact  of  time  on  negotiations.  A 
negotiator  usually  has  a  hard  time  deadline  before  which  the  negotiation  has  to  end.  In 
the  Navy  detailing  system  a  Sailor  has  a  certain  detailing  time  window,  which  is  usually 
three  months.  After  that,  if  the  Sailor  has  not  located  a  job  he  will  be  assigned  to  a  duty 
station  by  a  detailer.  Similarly,  a  Command  must  find  a  Sailor  to  fill  a  job  before  a 


1 A  negotiator  could  quit  the  negotiation  because  an  agreement  is  reached  in  another  negotiation  thread. 

2  The  strategies  of  players  constitute  a  perfect  Bayesian  equilibrium  if  given  the  strategies  of  the  other 

players;  a  player  cannot  obtain  strictly  better  profit  on  expectation  in  each  subgame  by  deviating  to 

another  sequential  strategy. 
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certain  time,  which  is  usually  before  the  current  job  incumbent  leaves.  With  less 
remaining  negotiation  time,  a  negotiator  is  more  pressed  to  reach  an  agreement  and  to 
concede.  But  a  negotiator  cannot  wait  until  the  last  moment  to  concede  because  the  time 
is  also  valuable.  The  time  spent  on  negotiation  should  be  reasonable  with  respect  to  the 
value  of  the  agreement  that  is  reached.  A  negotiator  who  persistently  holds  to  the  price 
risks  losing  the  negotiation  opponent  because  the  opponent  may  find  another  partner 
during  the  process.  There  can  be  other  factors  that  also  impact  the  negotiation  strategy 
such  as  the  available  resources.  Actually  the  remaining  time  can  be  viewed  as  one  type  of 
resource.  The  same  approach  can  be  used  to  model  the  impact  of  other  resources,  if 
there  are  any,  and  the  decision  can  add  to  the  decision  based  on  the  time-dependent 
strategy. 

In  the  time-dependent  approach,  time  is  the  predominant  factor  used  to  decide 
which  proposal  to  offer  or  accept  next.  For  the  buyer  in  a  negotiation  thread  di,  the 
proposal  to  offer  or  accept  is  within  the  interval  ( mim ,  maxi),  where  max,  is  the 
reservation  price  of  the  buyer  in  thread  di,  and  mim  is  the  lower  bound  of  a  valid  offer 
(we  can  reasonably  assume  mim  =  o).  For  a  seller  mim  will  be  the  reservation  price  and 
maxi  is  the  upper  bound  of  a  valid  offer.  Initially  a  negotiator  offers  the  most  favorable 
value  for  herself.  If  the  proposal  is  not  accepted,  a  negotiator  concedes  with  time 
proceeding  and  moves  toward  the  other  end  of  the  interval.  The  pace  of  concession 
depends  on  the  negotiation  strategy  and  is  characterized  by  a  function  a,- of  time.  The 
value  x‘h  to  be  offered  by  a  buyer  and  the  value  x[  to  be  offered  by  a  seller  at  time  t,  t  e 
[o,  T  - 1],  are  as  follows: 


x'b  =  rnina  +«„(/)  (maxa  -  mina )  ( 1) 

x‘a  =mirib  +  (1  -  aa  (t))  ( maxb  -  mitib).  (2) 


The  buyer  accepts  an  offer  x‘s  from  negotiator  b  at  time  t  if  it  is  not  worse  than  the 
offer  he  would  submit  next  time  (i.e.,  x'+1  >x'h ).  Similarly  the  negotiator  b  accepts  an 
offer  x‘a  at  time  fif  x£+1>x' . 


The  time-dependent  function  can  be  defined  by  a  family  of  polynomial  functions^ 


(3) 


3  Alternatively  we  can  also  use  the  exponential  functional  family,  and  define  ai  (f)  =  e"  x" .  These  two 
families  are  similar  in  their  functionality  except  that  their  sensitivity  to  the  change  of  time  is  different  with 
different  ft.  For  the  same  big  value  of  P,  the  polynomial  function  concedes  faster  at  the  beginning  than  the 
exponential  one;  then  they  behave  similarly.  For  a  small  value  of  /?,  the  exponential  function  waits  longer 
than  the  polynomial  one  before  it  starts  conceding  (Faratin  et  al.,  1998). 
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The  constant  J3>  o  determines  the  concession  pace  along  with  time,  or  the  convexity 
degree  of  the  curve  of  proposals  (see  Figure  l).  By  varying  / 3  a  wide  range  of  negotiation 
strategies  can  be  characterized.  Two  sets  of  (i  can  be  identified  to  characterize  two 
classes  of  strategies:  Boulware  with  /?  <  l  and  Conceder  with  f3>  l.  With  a  Boulware 
strategy  a  negotiator  tends  to  maintain  the  offered  value  until  the  time  is  almost 
exhausted,  then  she  concedes  to  the  reservation  price  quickly.  With  a  Conceder  strategy 
a  negotiator  goes  to  the  reservation  price  rapidly  and  early.  Figure  l  shows  the  change  of 
offers  with  time  in  the  two  strategy  classes.  Which  strategy  to  use  depends  on  how  much 
a  negotiator  values  the  time  and  the  expectation  of  the  opponent’s  strategy.  An 
impatient  negotiator  wants  to  reach  a  deal  earlier  and  is  more  likely  to  follow  the 
Conceder  strategy.  If  a  negotiator  expects  the  opponent  to  be  a  conceder,  she  will  tend  to 
apply  a  Boulware  strategy. 


Figure  1.  Offer  curves  with  different  p. 

The  time-dependent  strategy  is  intuitive  and  simple,  and  has  proved  useful  in  real 
applications  (Faratin  et  al.,  1998).  The  shape  of  the  curve  of  concession,  or  the 
parameter  /?,  is  what  differentiates  the  strategies  of  a  negotiator.  The  disadvantage  of  the 
approach  is  that  the  real-time  information  in  the  negotiation  is  not  used.  Once  /?is 
chosen,  the  offer  curve  is  pre-determined.  But  the  bilateral  negotiation  based  on  an 
alternating-offers  protocol  is  a  sequential  interactive  process.  The  information  that  has 
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been  revealed  by  the  opponent  in  the  negotiation  can  be  useful  in  making  subsequent 
decisions.  This  consideration  is  included  in  the  Bayesian-learning  strategy. 

Bayesian- learning  Strategy 

In  the  Bayesian  learning  approach  a  negotiation  agent  uses  the  Bayesian  framework 
to  update  prior  knowledge  and  belief  about  the  environment  and  other  agents  based  on 
the  messages  that  have  been  exchanged  previously  in  the  negotiation  and  domain 
knowledge.  Based  on  an  increasingly  accurate  belief,  the  negotiator  can  make  better 
sequential  decisions  in  the  negotiation. 

Let  the  possible  type  of  the  opponent  be  in  the  collection  {hj}nM .  As  we  have  defined 

in  Li,  Giampapa,  &  Sycara  (2003a),  the  type  of  a  negotiator  is  the  private  information 
held  by  the  agent  that  impacts  the  negotiation  outcome.  The  reservation  price  is  the  type 
of  an  opponent.  The  prior  belief  is  the  probability  that  the  opponent  has  a  type  hj  and  is 
denoted  by  P{hj),j  =  1, ...,  n.  The  domain  knowledge  attaches  a  probability  P(e  I  hj)  to 
every  possible  action  e  of  the  opponent  conditional  on  the  type  hj.  An  opponent  proposal 
action  can  be  viewed  as  a  signal  of  the  opponent’s  type.  Given  the  encoded  domain 
knowledge  in  the  form  of  conditional  probabilities  and  the  signal  e  given  by  the 
opponent,  a  negotiator  can  use  the  standard  Bayesian  updating  rule  to  revise  her  belief 
about  the  opponent’s  type: 


P(hj\e) 


P(/q)P(el/q) 
ELi  P(e\hk)P(hkY 


(4) 


Given  her  revised  belief  the  negotiator  can  apply  its  decision  rule  to  make  a  proposal 
or  respond  to  an  offer.  The  decision  rule  can  be  a  simple  strategy,  for  example,  (for  the 
buyer)  to  propose  a  price,  which  is  10  percent  below  the  estimated  reservation  price  of 
the  seller.  Or  it  can  be  a  solution  to  an  optimization  problem,  which  provides  decision 
heuristics,  for  example,  to  make  a  proposal  that  maximizes  the  expected  utility 
assuming  the  negotiation  ends  next  period.  Then  the  proposal  of  the  buyer  at  period  t  is 
the  solution  of  argmaxxFt  (x)  ( v  -  x),  where  v  is  the  value  of  the  seller,  Ft  (x)  is  the 
probability  that  the  seller’s  reservation  price  is  less  than  x  based  on  the  revised  belief  at 
period  t.  The  decision  of  the  buyer  trades  off  between  the  probability  of  the  proposal 
being  rejected  and  the  profit  if  the  proposal  is  accepted.  The  solution  suggests  that  the 
next  proposal  xt  satisfies 


v  —  xt  =  — 


F(xt) 

F'(xtY 


(5) 


The  domain  knowledge  can  be  very  specific  and  confirmative,  for  example,  “in  our 
business  a  seller  usually  offers  a  price  17  percent  above  the  reservation  price.”  It  can  also 
be  simple  and  “natural”  (and  cannot  be  called  “domain”  knowledge  anymore);  for 
example,  a  seller  will  not  offer  a  price  that  is  lower  than  her  reservation  price.  While 
specific  domain  knowledge  allows  efficient  update  of  the  prior  belief,  domain  knowledge 
is  hard  to  identify  and  acquire.  It  also  requires  discretization  of  the  type  space  to  apply 
the  Bayesian  framework,  as  is  shown  in  Equation  4.  The  “natural”  knowledge  does  not 
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help  in  the  modeling  and  updating  of  the  prior  belief  as  much  as  specific  domain 
knowledge,  but  it  is  easy  to  acquire  and  may  not  need  a  discrete  type  space.  For 
example,  let  Ft(x)  be  the  current  belief  at  period  t  on  the  probability  that  the  reservation 
price  of  a  seller  is  less  than  x,xe  [x,  x  ].  If  the  seller  next  proposes  a  price  z  e  [x,  x  ], 
then  the  belief  can  be  revised  to  Ft+1(.)  based  on  the  knowledge  that  a  seller’s  reservation 
price  is  always  less  than  her  proposal.  Then  the  new  belief  is: 


Ft+i(x ) 


T$j  for  x  €  [x,  z } 
Ft+i(z)  forx€(z,x]. 


Both  the  belief  update  method  (6)  and  the  decision  function  (5)  have  been  applied  in 
previous  work  in  negotiations  (Li  &  Tesauro,  203;  Tesauro,  2002),  and  in  other 
problems  such  as  bidding  in  double  auctions  (Gjerstad  &  Dickhaut,  1998;  Tesauro  & 
Bredin,  2002).  They  can  be  used  to  build  a  basic  Bayesian-learning  negotiation  strategy 
if  no  other  domain  knowledge  or  more  efficient  decision  rule  is  available. 


Synchronized  Multi-threaded  Negotiations 

In  a  synchronized  multi-threaded  negotiation  process  a  negotiator  participates  in 
multiple  bilateral  negotiation  threads  with  different,  simultaneous  negotiation 
opponents.  The  negotiator  can  reach  an  agreement  in  at  most  one  of  these  threads,  and 
is  aware  of  all  the  threads  at  the  beginning  of  the  process.  From  one  thread’s 
perspective,  the  other  threads  are  outside  options.  The  reservation  utility  that  the 
negotiator  should  set  in  one  thread  is  equal  to  the  expected  utility  from  all  other  threads. 
The  other  threads  form  a  synchronized  multi-threaded  negotiation  with  one  less  thread 
than  the  original  process. 

As  we  have  explained  in  Li  et  al.  (2003a),  with  a  multi-threaded  negotiation  it  is 
reasonable  to  assume  that  if  any  agreement  is  reached,  the  agreement  is  signed  with  the 
most  competitive  opponents  among  all  opponents  of  the  threads.  For  a  buyer  the  seller 
Si  in  thread  d,  is  more  competitive  than  the  sellers  in  other  threads  if  s,  can  give  more 
utility  to  the  buyer  (i.e.,  y,  =  vl  -  n  is  greater  than  ijj,  dj  e  D\di,  where  n is  the  reservation 
price  the  seller  in  thread  d,,  and  D  =  {dx,...,  dN}  is  the  collection  of  threads).  The  amount 
iji  is  the  maximum  utility  that  the  buyer  can  achieve  from  the  negotiation  thread  d,. 
Because  the  buyer  does  not  know  the  reservation  price  of  a  seller,  he  also  does  not  know 
the  maximum  utility  in  a  thread.  Based  on  the  prior  belief  F  (•)  on  the  reservation  price 
of  a  seller,  the  negotiator  can  derive  the  probability  distribution  of  the  maximum  utility. 
From  the  probability  distribution  of  the  maximum  utility  in  every  thread,  the  probability 
distribution  of  the  highest  and  second  highest  maximum  utility  can  be  calculated.  Let  G, 
(y)  denote  the  probability  of  the  maximum  utility  in  thread  d,  being  less  than  y.  Let  G1 
(y)  and  G2  (y)  be  the  probability  distribution  of  the  highest  and  second  highest 
maximum  utilities.  These  probabilities  can  be  calculated  by  the  following  formulas: 

Gi(y)  =  Pr(vf  -  r,  <y)  =  Pr(r,  >  v,  -  y)  =  F(v,  -  y) 
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G\y)  =  II  Gi(y) 

didD 

G2(y)  =  G1(y)  +  't(  1-G*(y))  n  GM- 

i=l  dj€D\di 

The  corresponding  probability  density  functions,  or  the  derivatives  of  these 
(cumulative)  probability  distribution  functions,  are  as  follows: 

gd(y)  =  -f(vd-y) 

g\y)  =  9i{y)  II  GAy) 

di€D  dj€  D\dj 

92{y)  =  g1(y)  -J2di(y)  II  Gj{y)  +  £(1  -  G*(y))[  9j(y)  II  Gm(y)] 

i=l  djdD\di  i=l  djdD\di  dmdD\{di,dj} 

Four  heuristic  approaches  are  provided  to  estimate  the  expected  utility  OU(D )  from 
a  multi-threaded  negotiation  composed  by  the  threads  IX Li,  Giampapa,  &  Sycara, 
2003a): 

•  Conservative  estimation.  The  utility  of  the  buyer  is  equal  to  the  expected  second 
highest  maximum  utility.  This  approach  ignores  the  further  concession  of  the 
winning  seller  in  the  continued  single-threaded  bargaining  process  after  he/she 
outbids  the  other  opponents.  The  expected  utility  is  calculated  by 

OU  =  /  yy2(y)dy 
Jo 

where  y  is  the  upper  bound  of  the  possible  utility  that  the  negotiator  can  achieve. 
If  the  lower  bound  of  an  acceptable  price  for  a  seller  is  c  and  the  upper  bound  of  a 
buyer’s  valuation  is  v ,  then  y=v-  c. 

•  Medium  estimation.  Assume  the  continued  single-threaded  bargaining  ends  at  the 
middle  point  between  the  buyer’s  and  the  winning  seller’s  reservation  price,  if  the 
buyer’s  reservation  price  is  higher  than  the  winning  seller’s. 4  Then  the  expected 
utility  is  the  average  of  the  expected  highest  and  second  highest  maximum  utility. 

OU  =  {J\g2(y)dy  +  yg1(,y)dy)/2 


4  If  the  buyer’s  reservation  price  is  lower  than  the  seller’s,  there  is  no  “zone  of  agreement”  and  the 
negotiation  will  fail. 
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In  this  estimation  we  do  not  consider  the  probability  that  the  negotiation  may  fail 
even  if  an  agreement  is  actually  desirable  for  both  parties.  This  is  because  in  a 
negotiation  model  with  incomplete  information,  negotiators  are  not  willing  to 
reveal  their  reservation  prices  but  expect  the  concessions  of  the  other.  This 
inefficiency  is  considered  in  the  approach  of  uniform  approximation. 

•  Uniform  approximation.  Previous  research  has  established  an  optimal  bargaining 
result  between  a  buyer  and  a  seller  based  on  game  theoretic  analysis  when  both 
parties’  reservation  prices  follow  uniform  distributions  (Fudenberg  &  Tirole,  1991; 
Myerson  &  Satterthwaite,  1983).  Based  on  this  result,  an  agreement  occurs  if  and 
only  if  the  buyer’s  valuation  exceeds  the  seller’s  cost  by  at  least  one-fourth,  if  both 
parties’  reservation  prices  distribute  uniformly  on  [o,  1].  In  other  words,  an 
agreement  cannot  be  reached  if  the  buyer’s  valuation  is  less  than  the  seller’s  cost  plus 
one-fourth  of  the  maximum  difference  between  the  buyer’s  valuation  and  the  seller’s 
cost.  We  can  approximate  the  probability  distributions  of  negotiators’  types  by 
uniform  distributions  and  apply  this  result  to  calculate  the  probability  of  reaching  an 
agreement.  In  the  heuristic  we  assume  an  agreement  cannot  be  reached  in  the 
continued  single-threaded  negotiation  between  the  buyer  and  the  winning  seller  if 
the  maximum  utility  of  the  winning  seller  is  less  than  a  quarter  of  the  highest 
possible  utility  y .  In  this  case  the  buyer  achieves  the  second  highest  maximum 
utility,  which  is  the  reservation  utility  of  the  buyer  in  the  continued  single-threaded 
negotiation.  If  an  agreement  is  reached  in  the  single-threaded  negotiation,  it  is 
reasonable  to  assume  that  it  is  at  the  middle  point  between  both  parties’  reservation 
prices.  Therefore  in  this  case  the  buyer  achieves  the  medium  of  the  highest  and  the 
second  highest  maximum  utility. 

OU  -  yg  ^dy  +  yg  — —  [y  g\y)dy+  [*  yg2(y)dy(  1-  f  g\y)dy). 

2  Jy/4  JO  Jy/4 

•  Learning.  Learn  the  probability  of  reaching  an  agreement  and  the  distribution  of 
agreements  based  on  the  previous  negotiations  (Sycara,  1993).  The  result  of  learning 
is  represented  by  x  (Rb,  Rs), the  expected  agreement  on  the  price  from  the  negotiation 
when  the  buyer’s  and  seller’s  reservation  prices  are  Rb  and  Rs  respectively.  Given  the 
probability  distribution  of  the  opponent’s  reservation  price,  a  negotiator  can 
calculate  the  expected  utility  of  the  negotiation  based  on  the  result  of  learning.  If  the 
seller  s,  in  the  thread  d,  is  the  winning  seller,  then  the  probability  distribution  of  the 
reservation  price  is  F(c)n d  gD  (1  -  w  -  v,.  +  c)) ,  where  the  product  is  the  probability 

that  no  other  thread  dj  has  the  maximum  utility  vj  -  Cj  greater  than  the  maximum 
utility  Vi  -  a  in  thread  du  Then  the  expected  utility  from  a  multi-threaded  negotiation 
can  be  calculated  by 

OU  =  [  (vd~x(vi,c))  n  (!  ~  F{vj-Vi  +  c))dF{c) 

d,€D  -  dj€D\dt 
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If  negotiators  use  the  time-dependent  strategy  and  the  parameter  /?is  chosen 
randomly  with  the  mean  equal  to  l,  then  we  expect  negotiators  to  concede 
constantly  on  aggregation.  Then  the  result  of  learning  is  expected  to  be  close  to  the 
result  of  negotiation  when  ft  =  l  for  both  negotiators.  Let  the  reservation  prices  of 
the  buyer  and  the  seller  be  v  and  c  respectively.  Then 


(6) 


assuming  the  upper  bound  of  an  offer  is  l  and  the  lower  bound  is  o.  This  formula 
can  be  derived  from  Figure  2  that  shows  the  offer  curves  of  the  negotiators  with  / 3 
=  1  for  both  parties. 

Dynamic  Multi-threaded  Negotiations 

In  the  Navy  detailing  process  the  application  period  for  a  position,  or  search  period 
for  filling  a  job,  lasts  for  some  months.  During  that  period  potential  partners  are 
discovered  sequentially  and  new  negotiations  are  launched  dynamically.  For  an  ongoing 
negotiation  thread  the  outside  options  not  only  include  the  other  simultaneous 
negotiation  threads,  but  also  the  threads  that  may  be  launched  in  the  future. 

Considering  the  outside  options  in  the  future,  a  negotiator  must  decide  how  much  to 
offer  in  the  current  negotiation,  and  when  to  stop  searching  for  future  opportunities  and 
accept  an  offer  from  the  current  negotiation.  If  a  negotiator  knows  the  number  of 
outside  options  that  will  come,  and  the  value  of  the  opponent  in  each  outside  option, 
then  the  negotiator  can  apply  the  synchronized  multi-threaded  negotiation  model  to 
calculate  the  appropriate  reservation  price  in  each  thread.  But  in  the  Navy  detailing 
process,  neither  a  Command  nor  a  Sailor  is  sure  about  the  arrival  of,  and  the  opponents’ 
values  in,  future  outside  options.  The  reservation  utility  of  a  thread  is  the  expected 
utility  of  a  multi-threaded  negotiation— including  other  simultaneous  threads  and 
threads  launched  in  the  future— with  a  stochastic  thread  number  and  uncertain 
opponents. 
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Figure  2.  Offer  curves  with  p  =  1 

Following  the  usual  way  of  modeling  uncertain  arrival,  we  assume  the  arrival  of 
outside  options  follows  a  Poisson  process  (Lippman  &  McCall,  1976;  Lippman  &  McCall, 
1981;  Ross,  2000).  In  each  period  t,  t  =  o,...,  T  - 1,  there  is  probability  p  that  the  buyer 
finds  a  matching  alternative  and  launches  a  negotiation  thread.  The  granule  of  each 
period  is  small  enough  so  that  the  probability  that  there  is  more  than  one  arrival  in  one 
period  is  zero.  Again  the  value  v  of  a  seller  follows  the  probability  distribution  ®  (y)  = 
Pr{v  <  y),  where  ®  (y)  is  the  probability  that  a  seller’s  value  is  no  greater  than  y.  The 
reservation  price  Rs  of  a  seller  follows  the  prior  belief  F(x)  =  Pr(Rs  <  x).  A  Command  can 
evaluate  a  Sailor  by  checking  the  Sailor’s  background.  A  Sailor  also  knows  how  much  he 
prefers  a  job  by  acquiring  information  about  job  location,  responsibility,  etc.  But  how 
much  a  Command  values  a  Sailor  or  a  Sailor  values  a  Command  is  unknown  to  the 
Sailor  or  the  Command  respectively.  Therefore  a  negotiator  knows  the  value  of  an 
opponent  when  the  opponent  is  identified,  but  not  the  reservation  price  of  the  opponent. 

The  state  st  of  the  system  is  defined  as  the  number  of  threads  nt  and  the  value  of  each 
opponent  seller  Vd,St  =  {nt,{vd}"j= t} .  The  evolution  of  the  system  follows  the  rule 

j  {nt  +  1,  {vd}dLi  U  f }  if  an  opponent  with  value  v  arrives  at  period  t 
St+1  ~  1  st  if  no  arrival  at  period  t 

Let  OUt(st)  be  the  utility  that  the  negotiator  expects  from  the  dynamic  multi¬ 
threaded  negotiation  when  he/she  sees  the  system  state  st  at  period  t.  Following 
previous  thoughts  we  can  calculate  OU({n,{vd}nd=x}) ,  the  expected  utility  from  a 

synchronized  multi-threaded  negotiation  with  n  threads  and  the  opponent  in  thread  d 
valued  Vd,  d  =  1,...,  n.  The  transition  of  the  expected  utility  follows  the  rule 
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OUt(st)  =  (1  -p)OUt+i{st)  +  pEv[OUt+i({nt  +  1,  {i^SLi  U  v})], 
OUt-i(st-i)  =  OU(st-  i). 


(7) 


If  the  probability  of  arrival  at  each  period  is  p,  then  the  number  of  arrivals  >](m,  p) 
during  an  interval  with  length  x  follows  a  Poisson  distribution 

(pr  )" 

Ppr(n)=Pr( ij( r,p)=n)=epT  m  .  Equivalently  we  can  write  the  transition  of  the  expected 
utility  as 


OUt(st)  =  Er,[E{vdrdLV  jOU({nt  + 1,  {vd}ndU  U  M^}})  (8) 

where  r/  following  a  Poisson  distribution  Pp,t-i(  ),  and  v<i  independently  follows  the 
identical  distribution  .®  (•),  d  =  nt+  l, ...  ,nt+r/. 

To  set  the  reservation  price  of  a  thread,  the  negotiator  only  needs  to  calculate  the 
expected  utility  of  the  multi-threaded  negotiation,  which  does  not  include  that  thread, 
based  on  the  period  and  real-time  state.  Because  the  state  of  a  dynamic  multi-threaded 
negotiation  changes  from  period  to  period,  the  reservation  price  of  a  thread  may  also 
change  with  time_ 


OUt(st)  =  (1  -p)OUt+1(st)  +pOUt+1({nt  +  1,  {vd}^  U  ®})]  (9) 

The  expected  utility  of  a  dynamic  multi-threaded  negotiation  process  at  each  period 
with  each  state  can  be  calculated  backward  from  the  last  period  following  Equation  7  or 
forward  following  Equation  8.  If  there  are  at  most  N  threads  and  for  each  opponent 
there  are  M  possible  values,  then  the  number  of  possible  states  will  be  NM.  The 
computation  is  intractable  with  large  M.  To  simplify  the  computation  we  can 
approximate  the  result  by  having  the  opponent  value  instances  replaced  by  the  expected 
value  v  (i.e.,  which  is  equivalent  to 


OUt(st)  =  Er,[OU({nt  +  t,  MZi  U  (OI  (10) 

The  compromise  due  to  this  simplification  is  not  significant  if  the  expected  utility  of  a 
synchronized  thread  is  or  can  be  approximated  by  a  linear  function  of  the  opponents’ 
values. 


14 


Experiments 


Two  models  of  the  outside  options  have  been  presented,  the  synchronized  and 
dynamic  multi-threaded  negotiation  models,  and  four  heuristic  approaches,  the 
conservative  estimation,  the  medium  estimation,  the  uniform  approximation,  and 
learning  approach,  to  estimate  the  expected  utility  in  a  multi-threaded  negotiation.  By 
combining  different  outside  option  models  and  estimation  approaches,  we  can  have 
eight  decision  models  for  bilateral  negotiations  in  the  Navy  detailing  process.  In  this 
section  we  provide  experiments  to  illustrate  the  different  models  in  the  solution 
framework  and  the  performance  results  based  on  the  different  models.  We  used  the 
time-dependent  strategy  as  the  strategy  in  a  single-threaded  negotiation  because  it  is 
simple  to  compute.  In  the  solution  framework  that  we  have  presented,  the  reservation 
utility  is  an  important  system  variable  that  decides  the  reservation  price,  which  impacts 
the  offer  curve  based  on  a  specific  negotiation  strategy.  In  Section  3.1  we  show  how  the 
reservation  utility  of  a  negotiation  thread  evolves  with  time  and  the  change  of  outside 
options  in  the  synchronized  and  dynamic  multi-threaded  negotiation  models.  We  then 
show  the  impact  of  outside  options  on  the  negotiation  strategy  by  showing  the  offer 
curves  adjusted  by  the  reservation  prices,  compared  with  the  original  basic  offer  curve 
without  considering  outside  options.  In  Section  3.2  we  compare  the  average  utility  of  a 
negotiator  when  she  (1)  does  not  consider  outside  options,  (2)  when  she  only  considers 
concurrent  outside  options  (i.e.,  the  synchronized  multi-threaded  negotiation  model), 
and  (3)  when  she  considers  both  concurrent  outside  options  and  future  arrivals  (i.e.,  the 
dynamic  multi -threaded  negotiation  model).  The  performance  results  based  on  different 
utility  estimation  approaches  are  also  compared  and  discussed. 

Reservation  utilities  and  offer  curves 

The  impact  of  outside  options  on  the  negotiation  strategy  is  illustrated  by  a  specific 
example.  In  this  example  the  negotiator  is  a  buyer.  The  negotiation  deadline  is  T  =  20. 
The  buyer  believes  the  reservation  price  of  a  seller  follows  a  uniform  distribution  on  the 
interval  [0,1].  The  value  of  the  item  of  a  seller  to  the  buyer  is  also  uniformly  distributed 
on  [0,1].  In  each  period  with  probability  p  >  o  a  new  seller  arrives  and  a  negotiation 
thread  is  created.  The  shape  of  the  offer  curve  defined  by  Equations  3  and  1  (Section  2. 

1)  is  determined  by  the  parameter  /?.  We  ran  multiple  experiments  withp  =  0.05,  0.10, 
0.15,  0.20,  0.25,  and  random  /?.  The  resulting  curves  followed  the  same  pattern  for  all 
instances.  We  show  the  resulting  curves  based  on  one  instance  with  p  =  0.2  and 
P  =  1.262727.  The  arrivals  of  outside  options  in  the  instance  are  illustrated  in  Figure  3. 
The  figure  on  the  left  shows  the  time  of  arrival  and  the  value  of  each  arrival,  the  figure 
on  the  right  shows  the  number  of  threads  in  each  period. 
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Time  r,me 


Figure  3.  Arrivals  and  number  of  threads. 

To  illustrate  the  evolution  of  the  reservation  utility  of  a  thread,  we  collect  the 
reservation  utility  of  the  first  thread  along  time  calculated  with  different  estimation 
approaches  and  models  of  the  outside  options.  Figure  4  shows  the  reservation  utilities 
grouped  by  the  estimation  approach  and  Figure  5  compares  the  reservation  utilities 
calculated  based  on  different  estimation  approaches  but  on  the  same  outside  option 
model.  The  expected  utility  of  a  dynamic  multi-threaded  negotiation  process  was 
calculated  with  the  approximation  formula  shown  in  Equation  10. 

Figure  4  shows  that  the  reservation  utility  based  on  the  synchronized  model  without 
expecting  future  arrivals  is  less  than  the  reservation  utility  calculated  based  on  the 
dynamic  model  where  the  probable  future  arrivals  are  also  taken  into  consideration. 5 
This  is  as  expected  because  in  the  dynamic  model  a  negotiator  does  not  only  see  the 
current  existing  outside  options,  but  also  foresees  the  outside  options  that  may  come  in 
the  future.  Although  the  number,  the  arrival  times,  and  opponents’  values  of  the  future 
outside  options  are  uncertain,  they  are  still  valuable  and  provide  opportunities  of 


5  There  is  a  region  in  which  the  reservation  utility  based  on  the  synchronized  model  is  slightly  higher  than 
the  one  based  on  the  dynamic  model.  This  is  because  of  the  learning  model  that  is  used.  Consider  the 
situation  where  there  is  only  one  thread  d,.  The  expected  utility  based  on  the  learning  model 

is  v,  — — — —  .  Now  assume  a  new  negotiation  opponent  comes,  and  the  value  of  the  opponent  is  v2  and 
l  +  vi-q 

the  reservation  price  is  c2.  The  estimation  model  suggests  that  the  expected  utility  from  the  two-threaded 
negotiation  is  v,  — — — — — Pr(v2  -  c2  >  v,  -  c, )  +  v,  — — — —  Pr(v,  -  c,  >  v2  -  c2 )  •  When  v,  is  much  greater 

1  +  V2  -  C2  "  “  1  +  Vj  -  C] 

than  v2,  v,  C;  may  be  less  than  v.  V|  11 '  even  if  v2  -  c2  is  greater  than  u,  -  c,.  Therefore  more  threads 

1+V2  — C2  l  +  Vj-C, 

do  not  necessarily  mean  higher  expected  utility.  Arrivals  with  very  low  values  may  actually  reduce  the 
expected  utility  calculated  by  the  learning  approach.  But  generally  we  can  say  that  the  expected  utility 
increases  with  the  number  of  threads  and  hence  the  expected  utility  based  on  the  dynamic  model  is  higher 
than  the  one  based  on  the  synchronized  model. 
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reaching  an  agreement  outside  the  current  negotiation  threads.  Therefore  the 
expectation  on  the  possible  future  outside  options  raises  the  lowest  utility  that  a 
negotiator  can  accept  in  a  current  negotiation  thread. 


Figure  4.  Reservation  utilities  grouped  by  estimation  approaches. 

Figure  5  shows  that  the  approach  of  medium  estimation  suggests  a  higher 
reservation  utility  than  the  other  approaches  and  gives  the  most  optimistic  estimation 
on  the  utility  from  outside  options.  The  medium  estimation  gives  a  more  optimistic 
estimation  than  the  conservative  estimation  because  in  the  latter  the  concession  of  the 
winning  opponent  in  the  continued  single-threaded  negotiation  is  ignored.  The  expected 
utility  based  on  the  medium  estimation  is  higher  than  on  the  uniform  approximation 
estimation  because  the  inefficiency  of  a  negotiation  with  two-sided  incomplete 
information  is  considered  in  the  latter  but  not  in  the  former.  The  medium  estimation 
also  suggests  a  higher  expected  utility  than  the  learning  approach  because  in  the  latter 
the  negotiation  outcome  in  the  continued  single-threaded  negotiation  is  not  compared 
with  the  second  highest  maximum  utility,  while  in  the  former  it  is  guaranteed  that  the 
negotiation  outcome  in  the  continued  single-threaded  negotiation  is  not  worse  than  the 
second  highest  maximum  utility. 

No  matter  which  estimation  approach  is  used,  Figure  5  shows  that  the  reservation 
utility  based  on  the  synchronized  model  (Section  2.2)  monotonically  increases  with  time 
because  the  number  of  threads  increases  with  time.  But  it  is  interesting  to  note  that  the 
reservation  utility  based  on  the  dynamic  model  (Section  2.3)  is  not  a  monotonic  function 


17 


of  time.  This  is  because  there  are  two  forces  that  drive  the  change  of  the  reservation 
utility:  time  and  concurrent  threads.  When  the  negotiator  approaches  the  deadline,  the 
possibility  to  have  new  arrivals  decreases  and  it  drives  the  reservation  utility  down.  On 
the  other  hand,  the  reservation  utility  would  increase  with  the  arrival  of  a  new 
negotiation  opponent,  especially  when  the  value  of  the  new  opponent  is  high.  From 
Figure  5  we  can  see  that  the  reservation  utility  has  different  sensitivity  to  the  change  of 
time  and  new  arrivals  based  on  different  estimation  approaches.  The  reservation  utility 
based  on  the  learning  approach  does  not  change  as  much  as  the  ones  based  on  other 
approaches  as  time  or  outside  options  vary.  No  matter  which  estimation  approach  or 
outside  option  model  is  used,  the  resulting  reservation  utility  with  consideration  of 
future  outside  options  is  higher  than  without  considering  the  future  outside  options. 


Re^ervatton  utilities  the  synchronized  model  Reserve  fcn  uunu*s  *lth  the  dynamic  model 
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Figure  5  Reservation  utilities  grouped  by  outside  option  models. 


Based  on  the  reservation  utility  OUd  of  thread  d  we  can  calculate  the  reservation 
price  of  the  buyer,  Rd=  Vd-  OUd,  where  Vd  is  the  value  of  the  seller  in  thread  d.  We 
compare  the  offer  curves  based  on  different  estimation  approaches  and  outside  option 
models  in  Figure  6.  The  model  noted  by  “Single”  is  the  model  without  considering 
outside  options.  In  that  model  the  reservation  price  Rd  is  equal  to  the  opponent’s  value 
Vd  and  the  offer  xt  in  period  t  is  calculated  by  xt  =  (t/T)lPVd  following  Equation  1. 
(Assume  the  lower  bound  minj  of  a  valid  offer  is  zero.)  Without  considering  outside 
options,  the  offer  increases  with  time  as  the  buyer  constantly  concedes  (with  changing 
pace).  But  with  a  synchronized  or  dynamic  model  the  buyer  may  proceed  (i.e.,  decrease 
the  offered  price  from  the  previous  one,  when  a  valuable  new  opponent  arrives).  The 
pace  of  concession  is  also  different  with  different  outside  options  models.  When  a 
valuable  seller  arrives,  the  buyer  may  proceed  by  asking  for  a  lower  price  than  the 
previous  offer  because  the  buyer  gets  more  optimistic  about  the  expected  utility  that  can 
be  received  from  the  outside  options.  In  the  dynamic  model  the  speed  of  concession  at 
some  time  may  be  higher  than  without  considering  outside  options  in  the  single  model, 
because  of  the  impact  of  both  the  increasing  time  pressure  on  reaching  an  agreement 
and  the  decreasing  hope  on  the  availability  of  future  outside  options.  The  offers  without 
considering  outside  options  are  higher  than  the  offers  with  considering  only  concurrent 
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negotiation  threads,  which  are  again  higher  than  the  offers  with  additional 
considerations  of  outside  options  that  may  come  in  the  future.  This  is  consistent  with 
the  observation  that  the  reservation  utility  based  on  the  synchronized  model  is  less  than 
the  one  based  on  the  dynamic  model. 


Tb»  offer  curve  Iti  conservative  esttmatkro  The  offer  curve  wth  medium  efetmiatim 


Figure  6.  The  offer  curves. 


Performance  Results 

In  this  section  we  examine  and  compare  the  average  utilities  that  a  buyer  obtains 
with  three  different  outside  option  models  and  four  different  estimation  approaches. 
The  three  outside  option  models  include:  (l)  the  “Single”  model  in  which  no  outside 
options  are  considered,  (2)  the  “Synchronized”  model  in  which  only  concurrently 
existing  negotiation  threads  are  considered  as  outside  options,  and  (3)  the  “Dynamic” 
model  in  which  the  outside  options  also  include  the  possible  uncertain  future  arrivals. 
The  four  estimation  approaches  include  the  conservative  estimation,  the  medium 
estimation,  the  uniform  approximation,  and  the  learning  approach.  In  the  experiments 
the  negotiation  deadline  T  =  20.  The  buyer  believes  the  reservation  price  of  a  seller 
follows  a  uniform  distribution  on  the  interval  [o,  1].  The  value  of  a  seller’s  item  is  also 
uniformly  distributed  on  [0,1].  The  probability  that  a  new  seller  arrives  in  a  period  is  p, 
and  p  takes  the  values  {0.05,  0.10,  0.15,  0.20,  0.25}.  The  parameter  /? in  the  time- 
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dependent  strategy  of  a  negotiator  is  chosen  randomly  so  that  with  even  probability  a 
negotiator  in  a  thread  is  a  conceder  (J3  >  l)  or  a  Boulware  (J3<  l).  If  a  negotiator  is  a 
conceder,  p1  follows  a  uniform  distribution  on  [o,  l].  If  a  negotiator  is  a  Boulware,  /?is  a 
random  variable  with  a  uniform  distribution  on  [o,  l].  For  each  arrival  probability,  we 
repeat  the  experiment  50  times  and  the  average  utility  of  the  buyer  is  calculated. 

Figure  7  is  composed  of  four  subplots.  Each  subplot  shows  the  average  utility  as  a 
function  of  the  arrival  probability  based  on  one  estimation  approach,  and  with  different 
outside  option  models.  The  figure  implies  that  for  all  estimation  approaches  and  outside 
option  models,  the  average  utility  increases  with  the  arrival  probability.  This  is  intuitive 
and  should  be  true  for  a  reasonable  negotiation  strategy.  A  higher  arrival  probability 
implies  more  options  on  expectation  and  should  result  in  better  outcome  for  the 
negotiator.  Figure  7  also  shows  that  the  average  utility  based  on  the  dynamic  model  is 
higher  than  the  one  based  on  the  synchronized  model,  which  again  brings  higher 
average  utility  than  the  single-threaded  model  in  which  no  outside  option  is  considered. 
This  verifies  the  effectiveness  of  the  outside  option  models  that  we  have  proposed. 

We  can  also  group  the  average  utilities  by  the  outside  option  model  and  compare  the 
performance  of  the  estimation  approaches.  The  information  is  shown  in  Figure  8.  The 
figure  shows  that  there  is  no  estimation  approach  that  dominates  the  others.  This  is 
because  the  performance  of  an  approach  depends  on  the  negotiators’  offer  curves.  If 
both  negotiators  tend  to  concede  quickly  (/?is  very  large),  an  optimistic  estimation 
approach  such  as  the  medium  approach  may  be  better.  On  the  other  hand  if  both 
negotiators  tend  to  hold  on  their  positions  (/?is  very  small),  the  conservative  estimation 
approach  may  be  better. 
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Figure  7.  The  average  utilities  grouped  by  estimation  approaches. 
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Discussion  and  Conclusion 


In  this  report  an  integrative  solution  is  provided  for  the  negotiation  decision 
problem  in  the  Navy  detailing  system  when  negotiators  (Sailors,  Commands)  face 
uncertain  and  dynamic  outside  options.  The  outside  options  influence  the  negotiation 
strategies  via  the  impact  on  the  reservation  prices.  The  solution  is  composed  of  three 
modular  models:  single-threaded  negotiations,  synchronized  multi -threaded 
negotiations,  and  dynamic  multi-threaded  negotiations.  The  single-threaded  negotiation 
model  provides  the  negotiation  strategy  given  the  reservation  price.  The  other  two 
models  calculate  the  reservation  price  based  on  the  model  of  outside  options.  The  model 
of  synchronized  multi-threaded  negotiations  considers  the  presence  of  concurrently 
available  outside  options  and  provides  an  approach  to  estimate  the  outcome  when  the 
threads  are  known.  The  model  of  dynamic  multi-threaded  negotiations  expands  this  last 
model  by  considering  the  uncertain  outside  options  that  may  come  dynamically  in  the 
future.  The  specific  solution  for  each  module  is  presented,  and  experimental  analysis  is 
provided.  The  results  show  that  the  utility  of  a  negotiator  improves  significantly  when 
she  considers  outside  options  than  when  she  does  not  consider  them,  and  when  she 
considers  the  dynamic  arrival  of  outside  options  than  when  she  only  considers  the 
concurrently  existing  negotiation  threads. 


Figure  8.  The  average  utility  grouped  by  outside  option  models. 

The  following  remarks  may  avoid  possible  confusions  in  understanding  the  model: 

•  We  take  an  artificial  intelligence  approach  instead  of  a  game  theoretic  or  economics 
approach  in  this  study  because  the  complexity  of  the  situation  does  not  allow 
rigorous  mathematical  analysis,  which  is  usually  preferred  by  economists.  The 
approaches  in  the  AI  field  are  different  from  the  models  in  economics  in  that  AI 
approaches  aim  to  provide  an  effective  heuristic  solution  to  general,  realistic  and 
complicated  situations,  that  are  not  amenable  to  a  rigorous  mathematical  analysis. 
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•  We  are  not  using  an  auction  mechanism.6  In  a  multi -threaded  negotiation  process, 
each  negotiation  thread  is  an  outside  option  of  other  threads.  But  in  an  auction, 
which  includes  all  candidates  as  bidders,  there  is  no  outside  option  for  the 
auctioneer. 

•  The  “reservation  price”  should  not  be  confused  with  the  “intrinsic  value”  of  an  item. 
The  reservation  price  from  a  buyer’s  point  of  view  is  the  highest  price  that  is 
acceptable  in  one  negotiation.  The  highest  acceptable  price  is  no  greater  than  the 
intrinsic  value.  When  there  are  outside  options,  the  highest  acceptable  price  may  be 
less  than  the  intrinsic  value. 

•  Outside  options  change  the  reservation  price  in  a  negotiation.  To  give  a  very  simple 
example,  if  a  buyer  knows  that  an  item  could  be  bought  from  a  seller  for  $20,  he/she 
will  not  buy  it  from  another  seller  for  more  than  $20,  although  he/she  values  the 
item  at  $25.  In  the  Navy  situation,  a  buyer  is  not  uncertain  about  the  availability  and 
quality  of  outside  options,  nor  is  the  exact  agreement  that  could  be  reached  in  other 
outside  negotiations  known.  An  appropriate  reservation  price  could  only  be  set  by 
estimating  the  utility  possibly  achieved  from  outside  options  with  reasonable 
heuristic  approaches,  which  we  have  provided  in  this  report. 

In  this  negotiation  solution  we  have  focused  on  the  negotiation  strategy  when  the 
negotiator  faces  uncertain  outside  options.  The  behavior  of  the  negotiation  opponents 
when  they  also  have  outside  options  was  not  explicitly  modeled.  The  outside  options  of 
an  opponent  are  unknown  to  the  negotiator  and  influence  the  reservation  price  of  the 
opponent.  Since  the  reservation  price  is  private  information,  the  outside  options  of  an 
opponent  can  be  taken  into  consideration  if  the  prior  belief  on  the  opponent’s 
reservation  price  also  includes  the  probabilistic  information  on  his/her  outside  options. 

In  this  report,  heuristic  solutions  for  the  negotiation  decision  problem  in  the  Navy 
detailing  process  are  proposed.  The  complexity  of  interactions  in  an  alternating-offers, 
bilateral  negotiation  with  two-sided  asymmetric  information  does  not  allow  the 
mathematical  analysis  of  optimal  strategies  with  general  settings  such  as  continuous 
type  space  and  general  probability  distribution  of  the  prior  belief,  even  without  outside 
options.  In  this  work  we  pursue  the  practical  effectiveness  of  the  solution.  We  have 
proposed  and  applied  negotiation  strategies  that  have  been  developed  by  others  and  us 
in  the  AI  field,  and  we  have  provided  reasonable  heuristic  approaches  to  set  appropriate 
reservation  prices  considering  outside  options.  Existing  results  from  economics  on  the 
optimal  solutions  of  simpler  models,  such  as  auctions  and  bilateral  negotiations  with 
uniform  distributions  of  the  prior  beliefs,  are  used  to  design  reasonable  heuristics  to 
solve  the  complicated  problem  in  our  model.  Because  of  the  heuristic  approach  that  we 
take  in  this  work,  extensive  simulations  are  needed  to  provide  rigorous  evaluation  on 
the  performance  of  different  models  and  approaches  in  different  environments.  We  do 
not  claim  that  the  heuristics  we  provide  in  this  report  are  complete.  Rather  they  reflect 
solutions  that  have  been  proven  useful  or  plausible.  Other  negotiation  strategies  and 
approaches  to  estimate  the  utility  from  a  multi-threaded  negotiation  can  be  plugged  in 


6  Even  with  auctions,  the  reservation  price  is  not  necessarily  equal  to  the  true  value  ( Auction  Theory,  Vijay 
Krishna,  Academic  Press,  2002). 
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the  solution  framework,  depending  on  the  assumptions  and  requirements  of  the 
underlying  application.  These  different  models  can  construct  a  library  of  decision 
functions  to  support  the  decision  of  negotiation  agents  in  different  environments. 

Bilateral  negotiations  are  a  useful  mechanism  to  realize  distributed  matching 
between  Sailors  and  Commands  that  do  not  get  matched  through  the  mass-matching 
market.  How  these  two  mechanisms  interact  with  each  other  depends  on  Navy  policy.  If 
a  Command  or  a  Sailor  has  to  accept  the  matching  result  from  the  mass-matching 
market,  then  the  bilateral  negotiations  can  be  regarded  as  outside  options  of  the  mass¬ 
matching  market.  How  much  to  bid  in  the  mass-matching  market  is  impacted  by  the 
reservation  utility  that  a  Command  or  a  Sailor  expects  to  obtain  via  bilateral 
negotiations.  Otherwise  if  a  Command  or  a  Sailor  can  reject  the  results  from  the  mass¬ 
matching  market,  then  the  bidding  decision  is  not  entangled  with  the  bilateral 
negotiation  process.  But  the  decision  to  accept  or  reject  the  outcome  of  the  mass¬ 
matching  mechanism  still  depends  on  the  expected  utility  from  the  bilateral 
negotiations. 
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